Virtual reality platform with haptic interface for interfacing with media items having metadata

ABSTRACT

Apparatuses, systems, and methods for interacting with media items in a virtual reality environment are provided. An apparatus is provided that includes processing circuitry configured to control a display of a virtual reality interface to render representations of media items returned from a query in the virtual reality environment. The processing circuitry is further configured to control the display to render metadata links between the selected metadata attribute and the representations of the media items that have the selected metadata attribute, detect, from a haptic interface, a haptic gesture to bundle and pull the metadata links to a foreground, and control the display to render representations of the media items having the selected metadata attribute in the foreground for selection.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/711,161 filed on Jul. 27, 2018, the entire contents of which arehereby incorporated herein by reference.

TECHNICAL FIELD

Example embodiments generally relate to information technologies and,more particularly, relate to apparatuses, systems, and methods forsearching and navigating through vast amounts of information to identifydesired information.

BACKGROUND

Conventional search and information navigation tools use antiquatedapproaches often based solely on text strings or hierarchicalfolder-based information storage. Such approaches are not intuitive forsearching and navigating and, therefore, require a degree of experienceand/or training to be efficient and effective in their use. Further,when attempting to interface with non-textual information, such as,images and videos, querying and navigation can be cumbersome anddifficult, requiring frequent scrolling, re-querying, or filtering toidentify desired information or content. As such, there is a need forimproved, non-conventional approaches to information querying andnavigation that are intuitive to users to assist in efficiently andeffectively identifying desired content.

BRIEF SUMMARY OF SOME EXAMPLES

According to some example embodiments, a system for interacting withmedia items in a virtual reality environment is provided. The system maycomprise processing circuitry, a virtual reality interface, and a hapticinterface device. The virtual reality interface may be operably coupledto the processing circuitry, and the virtual reality interface maycomprise a display and a motion sensor. The display may be configured torender a virtual reality environment. The motion sensor may beconfigured to detect motion of a user to modify a viewpoint into thevirtual reality environment rendered by the display. The hapticinterface device may be operably coupled to the processing circuitry.The processing circuitry may be configured to control the display torender representations of media items returned from a query. In thisregard, the media items may be associated with respective metadataattributes. The processing circuitry may be further configured to detectthe motion of the user via the motion sensor and modify the viewpointinto the virtual reality environment such that a representation of adesired media item is within view, detect, from the haptic interface, afirst haptic gesture to select a metadata attribute associated with thedesired media item, and control the display to render metadata linksbetween the selected metadata attribute and the representations of themedia items that have the selected metadata attribute. The processingcircuitry may also be configured to detect, from the haptic interface, asecond haptic gesture to bundle and pull selected metadata links to aforeground, control the display to render representations of the mediaitems having the selected metadata link in the foreground for selection,and receive, at the processing circuitry, a selection of the media itemsthat have the selected metadata links in response to detecting thesecond haptic gesture to bundle and pull the selected metadata links theforeground.

According to some example embodiments, an apparatus for interacting withmedia items in a virtual reality environment is provided. The apparatusmay include processing circuitry configured to control a display of avirtual reality interface to render representations of media itemsreturned from a query in the virtual reality environment. In thisregard, the media items may be associated with respective metadataattributes. The processing circuitry may be further configured to detectthe motion of the user via the motion sensor and modify the viewpointinto the virtual reality environment such that a representation of adesired media item is within view, detect, from the haptic interface, afirst haptic gesture to select a metadata attribute associated with thedesired media item, and control the display to render metadata linksbetween the selected metadata attribute and the representations of themedia items that have the selected metadata attribute. The processingcircuitry may also be configured to detect, from the haptic interface, asecond haptic gesture to bundle and pull selected metadata links to aforeground, control the display to render representations of the mediaitems having the selected metadata link in the foreground for selection,and receive, at the processing circuitry, a selection of the media itemsthat have the selected metadata links in response to detecting thesecond haptic gesture to bundle and pull the selected metadata links theforeground.

According to some example embodiments, a method is provided. The methodmay comprise controlling a display of a virtual reality interface torender representations of media items returned from a query in thevirtual reality environment. In this regard, the media items may beassociated with respective metadata attributes. The example method mayalso comprise detecting motion of a user via a motion sensor and modifythe viewpoint into the virtual reality environment such that arepresentation of a desired media item is within view, detecting, from ahaptic interface, a first haptic gesture to select a metadata attributeassociated with the desired media item, and controlling the display torender metadata links between the selected metadata attribute and therepresentations of the media items that have the selected metadataattribute. The example method may also comprise detecting, from thehaptic interface, a second haptic gesture to bundle and pull selectedmetadata links to a foreground, controlling the display to renderrepresentations of the media items having the selected metadata links inthe foreground for selection, and receiving, at the processingcircuitry, a selection of the media items that have the selectedmetadata links in response to detecting the second haptic gesture tobundle and pull the selected metadata links the foreground.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

Having thus described some example embodiments in general terms,reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and wherein:

FIG. 1 illustrates an example system for interacting with media items ina virtual reality environment according to an example embodiment;

FIG. 2 provides an example flowchart for performing a query andinterfacing with media item results according to an example embodiment;

FIG. 3 illustrates an example rendering of representations of mediaitems according to an example embodiment;

FIG. 4 illustrates an example rendering of representations of mediaitems with a finger pointer selector according to an example embodiment;

FIG. 5 illustrates an example rendering of representations of mediaitems with a device pointer selector according to an example embodiment;

FIG. 6 illustrates an example rendering of a palette device hand displayin a VR environment according to an example embodiment;

FIG. 7 illustrates example palette device hand display according to anexample embodiment;

FIG. 8 illustrates an example rendering of a palette device hand displaywith a finger pointer selector according to an example embodiment;

FIG. 9 illustrates an example rendering of representations of mediaitems with metadata links according to an example embodiment;

FIG. 10 illustrates an example rendering of representations of mediaitems with metadata links collected into a bundle according to anexample embodiment;

FIG. 11 illustrates an example rendering of representations of mediaitems with metadata links collected into a bundle and pulled accordingto an example embodiment; and

FIG. 12 illustrates an example method for interacting with media itemsaccording to an example embodiment.

DETAILED DESCRIPTION

Some example embodiments now will be described more fully hereinafterwith reference to the accompanying drawings, in which some, but not allexample embodiments are shown. Indeed, the examples described andpictured herein should not be construed as being limiting as to thescope, applicability, or configuration of the present disclosure.Rather, these example embodiments are provided so that this disclosurewill satisfy applicable legal requirements. Like reference numeralsrefer to like elements throughout.

As used herein the term “or” is used as the logical or where any one ormore of the operands being true results in the statement being true. Asused herein, the phrase “based on” as used in, for example, “A is basedon B” indicates that B is a factor that determines A, but B is notnecessarily the only factor that determines A.

As indicated above new and unconventional systems, apparatuses, andmethods are provided herein for interacting with media items that areresults of a query, in a virtual reality environment. In this regard,the example embodiments provided herein are superior to conventionaltechniques used for searching and navigating results of a query in theform of media items, such as images and videos. Prior arttwo-dimensional result environments have cumbersome and complex userinterfaces that require experience to efficiently utilize. Exampleembodiments improve upon such prior art techniques by permitting a userto intuitively interact with search results in a natural, intuituvemanner that leverages a virtual reality environment. According to someexample embodiments, metadata associated with media items may berendered and therefore is made “visible” to a user for intuitiveinteraction via gestures such as grasping metadata links between itemsin a VR environment. Example embodiments are superior to prior arttechniques because the example embodiments can allow users to visiblyunderstand and appreciate the metadata relationships between media itemsand navigate through the metadata relationships in a virtual realityenvironment to identify a target set of media items.

According to some example embodiments, a method and system thatleverages a haptic user interface in a VR environment for retrieving,displaying, modifying, and manipulating media items is provided. Themedia items may be derived from data sets of archives and databases.Example embodiments may allow users to query and browse archivedatabases of videos, images, two-dimensional and three-dimensionalscans, and other media items, and display the media items in a VRenvironment rendered in, for example, a head-mounted display. In thehead-mounted display, search results may be shown as a collection orconstellation that uses different parameters from the metadata of theitems to arrange and re-arrange the items in virtual space around theuser. Within the VR environment, the user may hold, for example, apalette device in one hand that displays a variety of object including adetailed version of a selected media item, media controls to manipulatethe selected media item, and metadata retrieved about that selectedmedia item. Using the other hand, the user may select and move items, aswell as, facilitate interaction with unique tangible metadata links inthe VR environment that may not been visualized or manipulated by othermeans. Such interactions with the VR environment, according to someexample embodiments, may be facilitated by use of controller gloves or agrasping/pointing tool the movement of which is detected and presentedin the VR environment.

Such example embodiments and others may be implemented in systems andapparatuses having a variety of architectures. One architecture is theexample system 10 shown in FIG. 1. The system 10 of FIG. 1 may include auser device 100, a network 106, and an archive 170. FIG. 1 shows some ofthe components of the user device 100. In this regard, the user device100 may include processing circuitry 101, a virtual reality (VR)interface 150, and a haptic interface device 160. The user device 100may include portions that reside in separate physical housings or may beintegrated into a single housing. For example, the processing circuitry101 may be a component of a computer system that that interfaces withthe VR interface 150, which may be separately housed. However, accordingto some example embodiments, the user device 100 may include a VRheadset that houses both the processing circuitry 101 and the VRinterface 150. According to some example embodiments, the hapticinterface device 160 may be embodied as or include a glove 161, pointingdevice 162, palette device hand display 163, or the like to interfacewith a user's hands, and thus may be other separate components of theuser device 100. Further, the haptic interface device 160 may alsoinclude a hand palette device that may be held in the hand of the userand interacted with as needed, similar to a tablet within the VRenvironment. The processing circuitry 101 may comprise a memory 102, aprocessor 103, a user interface 104, and a communications interface 105.The processing circuitry 101 may be operably coupled to the VR interface150 and the haptic interface device 160 as part of the user device 100.The processing circuitry 101 may be operably coupled to the network 106,which may be disposed external to the user device 100.

Further, according to some example embodiments, processing circuitry 101may be in operative communication with or embody, the memory 102, theprocessor 103, the user interface 104, and the communications interface105. The processing circuitry 101 may be configurable to perform variousoperations as described herein. In this regard, the processing circuitry101 may be configured to perform computational processing, memorymanagement, user interface control and monitoring, and manage remotecommunications according to an example embodiment. In some embodiments,the processing circuitry 101 may be embodied as a chip or chip set. Inother words, the processing circuitry 101 may comprise one or morephysical packages (e.g., chips) including materials, components or wireson a structural assembly (e.g., a baseboard). The processing circuitry101 may be configured to receive inputs (e.g., via peripheralcomponents), perform actions based on the inputs, and generate outputs(e.g., for provision to peripheral components). In an exampleembodiment, the processing circuitry 101 may include one or moreinstances of a processor 103, associated circuitry, and memory 102. Assuch, the processing circuitry 101 may be embodied as a circuit chip(e.g., an integrated circuit chip, such as a field programmable gatearray (FPGA)) configured (e.g., with hardware, software or a combinationof hardware and software) to perform operations described herein.

In an example embodiment, the memory 102 may include one or morenon-transitory memory devices such as, for example, volatile ornon-volatile memory that may be either fixed or removable. The memory102 may be configured to store information, data, applications,instructions or the like for enabling, for example, querying andnavigating media items in a VR environment in accordance with exampleembodiments. The memory 102 may operate to buffer instructions and dataduring operation of the processing circuitry 101 to support higher-levelfunctionalities, and may also be configured to store instructions forexecution by the processing circuitry 101. The memory 102 may also storeapplications for execution by the processing circuitry 101 in order tocarry out the functionality associated with each respective applicationor archives of media items as described herein. In this regard, forexample, rather than the archive 170 being accessible to the user device100 via the network 106, the archive 170 may, according to some exampleembodiments, be stored directly on the memory 102 locally.

As mentioned above, the processing circuitry 101 may be embodied in anumber of different ways. For example, the processing circuitry 101 maybe embodied as various processing means such as one or more processors103 that may be in the form of a microprocessor or other processingelement, a coprocessor, a controller or various other computing orprocessing devices including integrated circuits such as, for example,an ASIC (application specific integrated circuit), an FPGA, or the like.In an example embodiment, the processing circuitry 101 may be configuredto execute instructions stored in the memory 102 or otherwise accessibleto the processing circuitry 101. As such, whether configured by hardwareor by a combination of hardware and software, the processing circuitry101 may represent an entity (e.g., physically embodied in circuitry—inthe form of processing circuitry 101) capable of performing operationsaccording to example embodiments while configured accordingly. Thus, forexample, when the processing circuitry 101 is embodied as an ASIC, FPGA,or the like, the processing circuitry 101 may be specifically configuredhardware for conducting the operations described herein. Alternatively,as another example, when the processing circuitry 101 is embodied as anexecutor of software instructions, the instructions may specificallyconfigure the processing circuitry 101 to perform the operationsdescribed herein.

The communication interface 105 may include one or more interfacemechanisms for enabling communication with other devices external touser device 100, via, for example, the network 106, which may, forexample, be a local area network. In some cases, the communicationinterface 105 may be any means such as a device or circuitry embodied ineither hardware, or a combination of hardware and software that isconfigured to receive or transmit data from/to devices in communicationwith the processing circuitry 101. The communications interface 105 maybe a wired or wireless interface and may support various communicationsprotocols.

According to some example embodiments, the communications interface 105may be configured to communicate with external entities such as thenetwork 106. The network 106 may be any form of organizedinterconnection of communicating devices such as a personal areanetwork, a local area network, or a wide area network. The network 106may also embody combinations of sub-networks that include connectionsbetween the sub-networks. According to some example embodiments thenetwork 106 may include or incorporate the Internet.

The user interface 104 may be controlled by the processing circuitry 101to interact with peripheral devices that can receive inputs from a useror provide outputs to a user. In this regard, via the user interface104, the processing circuitry 101 may be configured to provide controland output signals to a peripheral device such as, for example, the VRinterface 150 and the haptic interface device 160, and receive inputsignals from a peripheral device. The user interface 104 may alsoproduce outputs, for example, to render a VR environment on the display151 of the VR interface 150. According to some example embodiments, theuser interface 104 may also operably couple to other user input oroutput devices such as, for example, a keyboard, mouse, touch screen,speaker, microphone, or the like.

As mentioned above, the processing circuitry 101 may be operably coupledto and control various user interface devices such as a VR interface 150and a haptic interface device 160. The VR interface 150 may include adisplay 151 and a motion sensor 152. The VR interface 150 may beembodied in various forms, including, for example, a VR headset. In thisregard, the display 151 may be disposed near the users eyes tofacilitate creation of an immersive visual presentation of a VRenvironment. In addition to the display 151, the VR interface 150 mayinclude other sensory outputs to the user, such as audio outputs viaspeakers or the like.

The VR interface 150 may also include a motion sensor 152. The motionsensor 152 may be an accelerometer or similar device for detecting orsensing motions performed by the user. In this regard, in exampleembodiments where the VR interface 150 is a headset, the motion sensor152 may be configured to detect movements of the user's head and thusprovide an input the processing circuitry 101 that can be used to modifya viewing direction into the VR environment based on the user's headmovement.

The haptic interface device 160 may be any type of the device thatincludes one or more sensors that can detect a gesture (e.g., a handmotion gesture) performed by a user. A gesture may be a sequence ofmovements that when considered together indicate an action to beperformed in the VR environment. In this regard, the haptic interfacedevice 160 may include a glove (e.g., one on each hand) with motionsensors that are configured to detect finger, hand, and arm motions thatcan be interpreted as gestures. The motion sensors within the glove maybe associated with joints in the user's fingers and arms to determine aposition of each joint. The gloves may also include location sensorsthat are configured to determine a position of the glove within a spacerelative to an origin or within a coordinate system. Further, asmentioned above, the haptic interface device 160 may also include apalette device that can be held and referred to while in the VRenvironment. The palette device may, according to some exampleembodiments, be configured to communicate interactions with the palatedevice to the processing circuitry 101 as a haptic interface device 160.The haptic interface device 160 may also include motion sensors, such asaccelerometers, that are configured to detect relative motion to therebypermit the processing circuitry 101 to determine positions of the hapticinterface device 160 (e.g., positions of the hands, possibly, inrelation to the palette device). The haptic interface device 160 mayprovide input signals to the processing circuitry 101 and the inputsignals can be analyzed to determine if a discernable gesture has beenperformed by the user. In this regard, the signals may be compared to adigital library of possible gestures (e.g., stored within the memory102) to determine a match. For example, the user may perform a pointinggesture (e.g., extended index finger and arm in a direction in front ofand to the upper left of the user's head) that may be interpreted as aselection operation of an item that is disposed in a space in the VRenvironment above and the to the left of the user's viewpoint within theVR environment. As such, due to the various motion sensors of the hapticinterface device 160, the haptic interface device 160 may facilitatemovement of, for example, a pointer or the like in a VR environmentbeing presented to the user via the display 151.

According to some example embodiments, the user device 100 may also haveaccess to an archive 170. The archive 170 may comprise a collection ofmedia items and associated metadata for each media item stored in avariety of manners, such as, in a database. As such, the archive 170 mayinclude a plurality of media items that may be linked or classifiedbased on shared attributes. The archive 170 may be searchable via aquery and media items, or portions of media items (e.g., thumbnails,clips, or the like), may be returned to the processing circuitry 101 asresults from the query for subsequent interaction as further describedherein. As shown in FIG. 1, the archive 170 may be accessible to theuser device 100 via the network 106 and may be stored, for example, onservers located remote from the user device 100. However, according tosome example embodiments, the archive 170 may be stored locally, forexample, in memory 102. Further, according to some example embodiments,the archive 170 may be stored in a distributed fashion where, forexample, portions of the media items (e.g., thumbnails, clips, etc.) arestored or copied into a first storage environment (e.g., memory 102 orfront-line query servers) and complete instances of the media items arestored in a second storage environment (e.g., deep storage servers).

The example system 10 and the processing circuitry 101 may be configuredto perform various functionalities described herein. In this regard, theflowchart of FIG. 2 illustrates an example method with variousoperations that the system 10 may be configured to perform as describedherein according to some example embodiments. The example method of FIG.2 will now be explained with support from the example renderedvisualizations of a related VR environment in FIGS. 3-11.

The example method may begin with the user querying a media archivedatabase at 200, which may involve formulation of a query by a user anddelivery of the query to an archive database (e.g., archive 170). Thequery may include one or more query parameters that may be used todetermine satisfying results from the archive 170. The query parametersmay associate directly with metadata fields for the media items in thearchive 170 (e.g., using a “contains” requirement), or the queryparameters may designate broader categories of media item attributes(e.g., titles beginning with designated characters and wildcards, arange of years for creation of media items, geographic regions of originfor media items, language used in media items, etc.) Regardless of thetypes and scope of the query parameters, the results of the query may bereturned and the processing circuitry 101 may be configured to cause adisplay of the search results at 202, via, for example, the display 151of the VR interface 150 which may render the results for the user.

In this regard, upon execution of the query, a plurality of results maybe rendered that satisfy the query. According to some exampleembodiments, satisfaction of the query may be associated with asimilarity rating and, as set by a user, only results to that meet orexceed a threshold similarity rating with the query may be returned andrendered as results. The results may be rendered, for example, in mannerthat permits the user to move or navigate through the results usinghaptic inputs in the form of, for example, gestures. In this regard, forexample, the user may move their head (as detected by the motion sensorsof a headset, such as, VR interface 150) or move their hands (e.g., in aswimming motion to move through and around the media item results asdetected by sensors of an enabled glove, such as, haptic interfacedevice 160) to move or navigate through the media items to identify oneof media items of interested in the result set.

Further, the results may be rendered as representations of the fullmedia item. In this regard, representations in the form of thumbnails(e.g., lower quality versions of the full image or lower resolutionversion of the full image) may be provided for still images and videoclips (e.g., shorter and lower quality versions of a video) orthumbnails may also be provided for video clips. According to someexample embodiments, a maximum threshold for the memory size of therepresentation may be set and, for example, for video clips that exceedthe maximum threshold size when rendered at a reduced resolution, therepresentation of the video clip may be a still image thumbnail or ashortened version of the video clip. The representations of the mediaitems may also be interactive, such that, a user may select or bringinto view a representation of a desired media item to cause retrieval ofadditional information about the media item, such as metadata. Forexample, if the media item is a still image, the user may select therepresentation of the media item to render a preview of a full-qualityversion of the still image for further inspection by the user.Additionally, for a video media item, the user may select therepresentation of the media item to cause a preview of the full video tobe played for inspection by the user. Further, according to some exampleembodiments, a representation of a media item may be selected and, inresponse, metadata associated with the media item may be displayed forfurther inspection by a user. As such, the representations of the mediaitems returned from a query may be dynamic and interactive to allow theuser to discover additional information or otherwise further investigatea media item result in a VR environment.

Additionally, according to some example embodiments, a query result maybe returned as a media item collection. In this regard, therepresentation of the media item that is rendered as a result in the VRenvironment may be representative of a plurality of underlying mediaitems that have been grouped. According to some example embodiments, theexample system 10 and the processing circuitry 101 may be configured topermit a user to move or navigate into the constellation within the VRenvironment to further investigate individual media items within thegroup while removing from rendering media items that are not within thegroup. Accordingly, the group as a whole or individual items within thegroup may be selectable by the user, via the user device 100, in the VRenvironment.

Accordingly, based on the foregoing, the user device 100 and theprocessing circuitry 101 may be configured to receive inputs from the VRinterface 150 and the haptic interface device 160 to change thearrangement of the representations of the media items, at 204, that werereturned from the query in the VR environment. The change in thearrangement of the representations of media items may occur in responseto, for example, navigation performed by the user through gesturesapplied within the VR environment as described above. As a result, theprocessing circuitry 101 may be configured to cause a new display of thesearch results to be provided. The new display of the representations ofthe media items may be displayed in a new arrangement due to navigationthrough the results in the VR environment. In this regard, updated, newdisplays of the representations of media items may be provided asfurther navigation takes place. As such, the example method mayrepeatedly return to 204 to receive further inputs and cause a newdisplay of the results at 206 until a user has navigated in a mannersuch that the arrangement of representations of the media items isdesirable.

In this regard, FIG. 3 illustrates an example rendering 300 of a VRenvironment with an arrangement of representations of media items thathave been returned from a query and moved via navigation to a desiredviewpoint to be presented to the user within the VR environment. In thisregard, the representations include representation 302 of an airline jet(object A), representation 304 of a bear (object B), representation 306of a bird (object C), representation 308 of a paper airplane (object D),representation 310 of a shoe (object E), representation 312 of a lion(object F), representation 314 of a propeller airplane (object G),representation 316 of a dog (object H), representation 318 of penguin(object I), and representation 320 of an animal skeleton (object J).These representations may be rendered in an array in the VR environment,such that the representations appear to the be disposed on an internalsurface of a sphere with the viewpoint at the center of the sphere.Further, the representations of media items may also be rendered asconstellations or groups where certain metadata attributes may causecertain media items to be grouped in a constellation within the VRenvironment.

At 208, the example method may include receiving an input from the userselecting a media item in the VR environment. In this regard, having setthe viewpoint in the VR environment, the user may select one of themedia items by performing a gesture to select the desired media item,which may be detected and acted upon by, for example, the processingcircuitry 101. The motions associated with the selection gesture may bedetected by the haptic interface device 160 and associated signals maybe provided to the processing circuitry 101 for analysis to determinewhich representation has been selected.

As shown in FIG. 4, the rendering 400 of the VR environment may bealtered to show a virtual hand and arm to assist the user in pointing tothe desired media item for selection. Further, by detecting theextension of the pointer finger of the hand via, for example, sensors ina glove, the processing circuitry 101 may cause the display 151 torender a virtual pointer finger 402 as well as, for example, and apointer beam 404. Such an example embodiment may be implemented with ahaptic interface device 160 that includes a glove. Alternatively, asshown in the rendering 500 of FIG. 5, a pointer-type device 502 may berendered with a pointer beam 504, again, to assist the user withselection of a media item in the VR environment. Such an exampleembodiment may be implemented with a haptic interface device 160 thatincludes a handheld controller. In either case, the user has selectedthe representation 302 of the airline jet in this example scenario.

In response to receiving the input selection of the desired media itemat 208, a display of the selected media item may be provided on, forexample, a palette device hand display 602 at 210. In this regard, asshown in the rendering 600 provided in FIG. 6, the palette device handdisplay 602 may be rendered in the foreground of the VR environment. Thepalette device hand display 602 may be a component of the hapticfeedback device 160 or may be virtually embodied. The palette devicehand display 602 may be moveable via user movement of the hapticinterface device 160 and may also be interacted with via the hapticinterface device 160. As more clearly shown in FIG. 7, the palettedevice hand display 602 may include a representation of the selectedmedia item 604 with the title 606 of the selected media item beingprovided with (e.g., below) the representation 604. Further, the palettedevice hand display 602 may also include a metadata section 608 that maybe populated with metadata attributes or categories associated with theselected media item. In the scenario being described, the representation302 (associated with an image of an airline jet) has been selected andtherefore an associated image is provided at 604. Additionally, in themetadata section 608, the metadata attributes or categories for thescenario are airplane 610, date 612, author 614, and format 616. It isnoteworthy that, while the rendering 600 shows a palette device handdisplay 602, any other type of interface may be displayed in place ofthe palette device hand display 602 that at least includes someidentifying information for the selected media item (e.g.,representation, title, etc.) and a listing of the metadata associatedwith the selected media item. Further, any attributes or categories ofmetadata may be shown in the metadata section 608 and, therefore, theoptions for metadata attributes or categories are not limited to thoseprovided here. Further, as shown in FIG. 2, a user may repeatedly selectrepresentations of media items to cause receipt of selection inputs at208, and, as a result, cause associated palette device hand displays tobe rendered at 210.

At 212, the example method may further include receiving an input viathe palette device hand display 602 to take an action with respect tothe selected media item. In this regard, a user may select, for example,the representation 604 on the palette device hand display 602 to causean action to be performed with respect to the selected media item. As aresult, at 214, the processing circuitry 101 may be configured to causean action to be taken with respect to the selected media item. Forexample, if the selected media item is a video clip, then the video clipmay be played within the VR environment (e.g., on the palette devicehand display 602) for the user. In another example, if the selectedmedia item is a three-dimensional model of an object, then the objectmay be rendered in the VR environment (e.g., on the palette device handdisplay 602) in a manner that permits a user to manipulate thethree-dimensional object to view the object from any viewpoint. Inanother example, if the selected media item is a still image, then theimage may be rendered in the VR environment (e.g., on the palette devicehand display 602) at maximum quality to permit the user to view detailsof the image and, for example, zoom into and pan to selected portions ofthe image. In another example, if the selected media item is an audiorecording, then the audio recording may be played for the user to hearthe recording. As such various actions may be taken based on the type ofmedia item that has been selected. Further, such action at 214 may berepeated each time an associated input is received as indicated at 212.

At 216, according to the example method, input from a user may bereceived to select metadata on the palette device hand display 602. Inthis regard, the user device 100 and the processing circuitry 101 may beconfigured to receive inputs from, for example, the haptic interfacedevice 160 indicating which of the metadata attributes or categories isbeing selected. As shown in the rendering 800 of the VR environment inFIG. 8, a virtual hand and arm may be rendered to assist the user inpointing the desired metadata item. Further, by detecting the extensionof the pointer finger of the hand, the processing circuitry 101 maycause the display 151 to render a virtual pointer finger 802 and apointer beam 804. Such an example embodiment may be implemented with ahaptic interface device 160 that includes a glove. Alternatively, apointer-type device (e.g., device 502) may be rendered with a pointerbeam, again, to assist the user with selection of a metadata item in themetadata section 608 of the palette device hand display 602. Such anexample embodiment may be implemented with a haptic interface device 160that includes a handheld controller. In either case, in the examplescenario, the user has selected the metadata attribute for airplane 610.Upon selection, the rendering of the selected metadata attribute orcategory may be boxed or highlighted to indicate that the selection hasbeen made.

In response to receiving the metadata attribute or category selection,the example method may further include causing visual metadata links tobe rendered between the palette device hand display 602 and thedisplayed search results based on the selected metadata at 218. In thisregard, based on the selection of the metadata attribute or category,the processing circuitry 101 may be configured to cause the display 151to render metadata links between the representations of media items thatshare the selected metadata and the selected metadata attribute orcategory on the palette device hand display 602. Further, eachrepresentation of a media item having the selected metadata may be boxedor highlighted to indicate the metadata relationship. Additionally, auser may repeatedly select different metadata attributes or categoriesat 216 to cause different visual connections to the representations ofmedia items to be rendered at 218.

With respect to the scenario being described, the user selected themetadata for airplane 610 on the palette device hand display 602. Asshown in the rendering 900 in FIG. 9, having selected the metadata forairplane 610, metadata links may be rendered between the metadata forairplane 610 and the representations of media items that have airplanemetadata, i.e., representation 302 of the airline jet, representation308 of the paper airplane, and representation 314 of the propellerairplane. In this regard, the representation 302 of the airline jet maybe marked by box 908 and linked to the metadata for airplane 610 bymetadata link 906, the representation 308 of the paper airplane may bemarked by box 912 and linked to the metadata for airplane 610 bymetadata link 910, and the representation 314 of the propeller airplanemay be marked by box 904 and linked to the metadata for airplane 610 bymetadata link 902.

At 220, the example method may further comprise receiving inputs (e.g.,from the haptic interface device 160) indicating that a collection ofthe metadata links, performed by a user, has occurred. In this regard, auser, as detected by the haptic interface device 160, may perform agrasping gesture to grasp the metadata links in the VR environment intoa bundle to indicate control over and selection of the media itemsassociated with the metadata links in the bundle. According to someexample embodiments, a user may be selective in the metadata links thatare collected and therefore some links may be collected while others mayremain out of the collection of links that are grasped. As shown in therendering 1000 of FIG. 10, the user's hand 1010 may be rendered as theuser collects each of the metadata links 902, 906, and 910 and graspsthe collection of links as a bundle.

At 222, the example method may further include modifying thepresentation of the media items associated with the collected metadatalinks. In this regard, in response to collecting the metadata links intoa bundle, the representations of media items associated with thecollected links may be presented, by the display 151, in a foreground ofthe VR environment as being selected from the results set. Moregenerally, the representations of media items associated with thecollected links may be selected or otherwise separated from the otherquery results for further actions in isolation. Further, according tosome example embodiments, the representations of media items associatedwith the collected links may be presented in the foreground of the VRenvironment as being selected from the results set after also a pullinggesture is performed and detected after the grasp of the metadata linksis detected, for example, by the haptic interface device 160.Alternative to the pulling gesture, a button press or other interactionmay be the triggering event that is detected by the haptic interfacedevice 160 to cause the representations of media items associated withthe collected links be selected or otherwise separated from the otherquery results for further actions in isolation. Similar to theoperations above, a user may release the collection of metadata linksand re-collect the metadata links at 220 repeated times thereby leadingto repeated modified presentations of media items at 222.

In this regard, with reference to the rendering 1100 in FIG. 11, theuser has performed a pulling gesture on the bundle of metadata links900, 904, and 908 as shown by the moved hand 1100 and the stretchedappearance of the metadata links. Upon detection of this pulling gestureof the collection, the representations 302, 308, and 314 may be causedto be brought to the foreground of the VR environment. Accordingly, themedia items underlying these representations may be deemed to beselected from the query result set. As an alternative to detected thepulling gesture, according to some example embodiments, a virtual orphysical button press may be detected, and in response to such a buttonpress, the representations 302, 308, and 314 may be caused to be broughtto the foreground of the VR environment.

Additionally, the example method may include receiving input from theuser on the palette device hand display 602 to save the selected mediaitems at 224. In this regard, the user may interface with the palettedevice hand display 602 to request that the selected media items orinformation associated with and identifying the selected media items besaved. In this regard, at 226, information associated with the selectedmedia items may be saved to a user profile. The user profile may definean instance of a person or login on a computer system. The informationassociated with media items may be saved in the form of a playlist forlater reviewing or playing by the user associated with the user profile.

Having described some example embodiments with reference to FIGS. 1 and2-11, FIG. 12 provides another flowchart of an example method that maybe performed by the user device 100 or the processing circuitry 101,possibly within the context of the system 10. In this regard, an examplemethod for interacting with media items is also provided. The examplemethod may include receiving a query at 1200 and controlling a displayof a virtual reality interface to render representations of media itemsreturned from the query in a virtual reality environment at 1210.Further, at 1220, the example method may include detecting motion of theuser from a motion sensor of the virtual reality interface and modifyinga viewpoint into a virtual reality environment such that arepresentation of a desired media item is within view. The examplemethod may further comprise, at 1230, detecting, from a hapticinterface, a first haptic gesture to select the representation of thedesired media item. Additionally, according to some example embodiments,the example method may include detecting, from the haptic interface,another (e.g., third) haptic gesture to select a metadata attributeassociated with the desired media item. At 1240, controlling the displayto render metadata links between the selected metadata attribute and therepresentations of the media items that have the selected metadataattribute. Additionally, the example method may include, at 1250,detecting, from the haptic interface, a second haptic gesture to bundleand pull the metadata links to a foreground. At 1260, the example methodmay comprise controlling the display to render representations of themedia items having the selected metadata attribute in the foreground forselection. Further, at 1270, the example method may include receiving,at the processing circuitry, a selection of the media items that havethe selected metadata links in response to detecting the second hapticgesture to bundle and pull the selected metadata links the foreground.

Additionally, according to some example embodiments, the example methodmay include controlling the display to render the metadata links asconnecting lines between the selected metadata attribute and therepresentations of the media items that have the selected metadataattribute. Additionally or alternatively, the example method may furtherinclude detecting the second haptic gesture to bundle the metadata linksas comprising a detection of a hand moving from an open position to aclosed position. Additionally or alternatively, the virtual realityinterface comprises a head-mounted display and the haptic interfacedevice comprises a glove. Additionally or alternatively, the examplemethod may further include controlling the display to render the desiredmedia item on a palette device hand display within the virtual realityenvironment. Additionally or alternatively, the example method mayinclude controlling the display to render a plurality of metadataattributes associated with the desired media item on the palette devicehand display within the virtual reality environment.

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseinventions pertain having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the inventions are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Moreover, although the foregoing descriptions and the associateddrawings describe exemplary embodiments in the context of certainexemplary combinations of elements or functions, it should beappreciated that different combinations of elements or functions may beprovided by alternative embodiments without departing from the scope ofthe appended claims. In this regard, for example, different combinationsof elements or functions than those explicitly described above are alsocontemplated as may be set forth in some of the appended claims. Incases where advantages, benefits or solutions to problems are describedherein, it should be appreciated that such advantages, benefits orsolutions may be applicable to some example embodiments, but notnecessarily all example embodiments. Thus, any advantages, benefits orsolutions described herein should not be thought of as being critical,required or essential to all embodiments or to that which is claimedherein. Although specific terms are employed herein, they are used in ageneric and descriptive sense only and not for purposes of limitation.

That which is claimed:
 1. A system for interacting with media items in avirtual reality environment, the system comprising: processingcircuitry; a virtual reality interface operably coupled to theprocessing circuitry and comprising a display and a motion sensor,wherein the display is configured to render a virtual realityenvironment and wherein the motion sensor is configured to detect motionof a user to modify a viewpoint into the virtual reality environmentrendered by the display; and a haptic interface device operably coupledto the processing circuitry; wherein the processing circuitry isconfigured to: control the display to render representations of mediaitems returned from a query, the media items being associated withrespective metadata attributes; detect the motion of the user via themotion sensor and modify the viewpoint into the virtual realityenvironment such that a representation of a desired media item is withinview; detect, from the haptic interface, a first haptic gesture toselect a metadata attribute associated with the desired media item;control the display to render metadata links between the selectedmetadata attribute and the representations of the media items that havethe selected metadata attribute; detect, from the haptic interface, asecond haptic gesture to bundle and pull selected metadata links to aforeground; control the display to render representations of the mediaitems having the selected metadata link in the foreground for selection;and receive, at the processing circuitry, a selection of the media itemsthat have the selected metadata links in response to detecting thesecond haptic gesture to bundle and pull the selected metadata links theforeground.
 2. The system of claim 1, wherein the processing circuitryis configured to control the display to render the metadata links asconnecting lines between the selected metadata attribute and therepresentations of the media items that have the selected metadataattribute.
 3. The system of claim 1, wherein the processing circuitry isconfigured to detect the second haptic gesture to bundle the metadatalinks as comprising a detection of a hand moving from an open positionto a closed position.
 4. The system of claim 1, wherein the virtualreality interface comprises a head-mounted display and the hapticinterface device comprises a glove.
 5. The system of claim 1, whereinthe processing circuitry is further configured to detect, from thehaptic interface, a third haptic gesture to select the representation ofthe desired media item.
 6. The system of claim 5, wherein the processingcircuitry is further configured to control the display to render thedesired media item on a palette device hand display within the virtualreality environment.
 7. The system of claim 6, wherein the processingcircuitry is further configured to control the display to render aplurality of metadata attributes associated with the desired media itemon the palette device hand display within the virtual realityenvironment.
 8. An apparatus for interacting with media items in avirtual reality environment, the apparatus comprising processingcircuitry configured to: control a display of a virtual realityinterface to render representations of media items returned from a queryin the virtual reality environment, the media items being associatedwith respective metadata attributes; detect motion of a user via amotion sensor and modify the viewpoint into the virtual realityenvironment such that a representation of a desired media item is withinview; detect, from a haptic interface, a first haptic gesture to selecta metadata attribute associated with the desired media item; control thedisplay to render metadata links between the selected metadata attributeand the representations of the media items that have the selectedmetadata attribute; detect, from the haptic interface, a second hapticgesture to bundle and pull selected metadata links to a foreground;control the display to render representations of the media items havingthe selected metadata links in the foreground for selection; andreceive, at the processing circuitry, a selection of the media itemsthat have the selected metadata links in response to detecting thesecond haptic gesture to bundle and pull the selected metadata links theforeground.
 9. The apparatus of claim 8, wherein the processingcircuitry is configured to control the display to render the metadatalinks as connecting lines between the selected metadata attribute andthe representations of the media items that have the selected metadataattribute.
 10. The apparatus of claim 8, wherein the processingcircuitry is configured to detect the second haptic gesture to bundlethe metadata links as comprising a detection of a hand moving from anopen position to a closed position.
 11. The apparatus of claim 8,wherein the virtual reality interface comprises a head-mounted displayand the haptic interface device comprises a glove.
 12. The apparatus ofclaim 8, wherein the processing circuitry is further configured todetect, from the haptic interface, a third haptic gesture to select therepresentation of the desired media item.
 13. The system of claim 12,wherein the processing circuitry is further configured to control thedisplay to render the desired media item on a palette device handdisplay within the virtual reality environment.
 14. The system of claim13, wherein the processing circuitry is further configured to controlthe display to render a plurality of metadata attributes associated withthe desired media item on the palette device hand display within thevirtual reality environment.
 15. A method comprising: controlling adisplay of a virtual reality interface to render representations ofmedia items returned from a query in the virtual reality environment,the media items being associated with respective metadata attributes;detecting motion of a user via a motion sensor and modify the viewpointinto the virtual reality environment such that a representation of adesired media item is within view; detecting, from a haptic interface, afirst haptic gesture to select a metadata attribute associated with thedesired media item; controlling the display to render metadata linksbetween the selected metadata attribute and the representations of themedia items that have the selected metadata attribute; detecting, fromthe haptic interface, a second haptic gesture to bundle and pullselected metadata links to a foreground; controlling the display torender representations of the media items having the selected metadatalinks in the foreground for selection; and receiving, at the processingcircuitry, a selection of the media items that have the selectedmetadata links in response to detecting the second haptic gesture tobundle and pull the selected metadata links the foreground.
 16. Themethod of claim 15, further comprising controlling the display to renderthe metadata links as connecting lines between the selected metadataattribute and the representations of the media items that have theselected metadata attribute.
 17. The method of claim 15, furthercomprising detecting the second haptic gesture to bundle the metadatalinks as comprising a detection of a hand moving from an open positionto a closed position.
 18. The method of claim 15, wherein the virtualreality interface comprises a head-mounted display and the hapticinterface device comprises a glove.
 19. The method of claim 15, furthercomprising detecting, from the haptic interface, a third haptic gestureto select the representation of the desired media item.
 20. The methodof claim 19, further comprising controlling the display to render thedesired media item on a palette device hand display within the virtualreality environment.